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one's  desktop,  on  the  basis  of  where  it  is,  or  where  one  put  it,  in  a 
well-learned,  familiar  space. 

A prototype  system  has  been  developed  that  uses  wall-sized,  full  color 
digital  television  with  synchronized  stereo  sound  to  create  a virtual 
spatial  world,  "Dataland,"  over  which  the  user  helicopters  via  joystick 
control.  Items  of  interest  seen  through  a graphics  "window"  can  be  zoomed 
in  upon  and  interactively  perused.  Data  types  include:  maps,  text,  book-like 
items,  letters,  photographs,  slides,  movies,  sound  and  television. 

Results  thus  far  suggest  that  the  user  quickly  learns  (on  the  order  of 
minutes)  to  navigate  about  such  a space,  and  readily  adopts  a spatial 
way  of  regarding  and  discussing  data. 

The  approach  of  managing  data  spatially  is  offered  not  as  an  alternative 
in  competition  with  managing  data  on  a symbolic  or  name  basis,  but  as  a 
complement  thereto.  This  manner  of  dealing  with  data  should  have  special 
appeal  for  that  class  of  user  for  whom  directness  and  immediacy  are  essential 
qualities  for  interaction.^  Economically,  there  is  increasingly  less  justi- 
fication that  the  potential'  richness  of  such  an  approach  to  managing  data 
need  be  foregone. 
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Prologue 


Such  startling  advances  and  cost  reduc- 
tions are  occurring  in  microelectronics 
that  we  believe  future  systems  will  not  be 
characterized  by  their  memory  size  or 
processing  speed  Instead,  the  human 
interface  will  become  the  major  measure, 
calibrated  in  very  subjective  units,  so 
sensory  and  personalized  that  it  will  be 
evaluated  by  feelings  and  perceptions 
Is  it  easy  to  use7  Does  it  feel  good7  Is  it 
pleasurable7 

It  is  this  interface  that  will  bring  com- 
puters directly  to  generals,  presidents  of 
companies,  and  six-year-old  children 
Thus,  we  ask  the  reader  of  this  report  to 
think  about  its  contents  in  two  ways.  One 
is  to  |udge  the  specific  value  of  spatial 
constructs  for  nonspatial  data.  The  other 
is  to  appreciate  the  general  intention  of 
making  sound,  visual,  and  tactile  inter- 
faces to  serve  conjointly  as  the  modes 
and  media  of  interaction  with  a comput- 
ing resource,  at  once  transparent  and 
ubiquitous.  Such  quality  is  no  longer  a 
luxury  but  a requirement. 


Nicholas  Negroponte 


The  Concept 


The  basis  ot  spatial  data  management  is 
accessing  a data  item  bv  going  to  where 
it  is  rather  than  referencing  it  bv  name 

Consider  what  occurs  when  you  retrieve 
a book  in  your  home  library  You  look  at 
the  left  hand  bookcase,  second  shelt 
from  the  top  say  and  scan  about  two 
thirds  down  the  shelf  tor  a certain  book 
spine  Thus,  the  book  is  located  not  by  its 
title  but  by  its  customary  site  m a space 
defined  by  othei  hooks  and  the  hook 
case  itself 

Consider  how  people  ordinarily  find 
items  on  thou  desk  tops  the  appoint 
ment  book  is  up  and  to  the  right,  the  tele- 
phone is  in  Hit'  lower  right  corner,  high 
priority  memos  are  kept  in  an  "in"  box 
immediately  to  the  left  of  the  desk  blotter, 
less  urgent  items  art'  in  the  tilt'  (older,  top 
middle  tit  desk,  and  so  forth 


1 lit'  per stm  wlit'  uses  this  desk  has 
organized  the  layout  ot  items  in  a mint' m 
less  systematic  way  Me  or  she  leteis  to 
them  constantly  throughout  the  working 
day  reaching  in  that  direction,  that  tar 
up  down  to  the  right  to  the  left  Through 
this  activity  a mental  image  of  the  layout 
of  the  desk  is  elaborated  in  the  "mind  s 
eye  Additionally,  through  constant  tac 
tile  interaction  with  the  items,  reaching  toi 
and  touching  them,  a 'motor  memory  of 
wtiere  things  ate  arises  vis  well  A script 
tor  the  act  ot  retrieval  becomes  encoded 
into  the  musculature,  vis  it  were,  viccord 
mg  to  where  the  item  is  located 


The  practical  importance  ot  this  spatial  It  is  surprising  how  pervasive  the  under - 
principle  of  organization  is  illustrated  lying  notion  of  spatiality  is.  oven  in  sym- 

wfien  someone  happens  to  disturb  our  bolic  modes  of  thought  Consider  a file  of 

familiar  arrangement  of  things  Perhaps  three  by-five  notecards.  organized 
some  well-meaning  soul  tries  to  tidy  up  alphabetically,  but  without  letter  tabs 

our  seemingly  chaotic  desk  top  Rather  showing  "Filed  under  the  letter  R"  trans- 

than  being  a benefit,  the  now  "well-  lates  into  a tactile  estimate  of  how  far 

organized”  desk  is  for  us  an  organize-  down  the  row  of  cards  we  must  reach 

tional  disaster  the  former  "messy  The  same  holds  true  for  a dictionary  with 

arrangement  was  a familiar  and  well  trav  out  thumb  indexes  When  we  want  to 
eled  ground  whereupon  we  could  go  look  up  a word  beginning  with  the  letter 

directly  to  what  we  wanted  Such  a loss  R.  we  "guesstimate”  where  we  should 
of  a familiar  spatial  layout  can  impress  open  the  pages  ot  the  book  to  begin  our 

upon  us  the  extent  to  which  we  take  this  search  Thus.  R is  somehow  a relative 

subtle  but  powerful  principle  of  orgamzn-  distance,  as  well  as  a letter  There  is 
tion  tor  granted  in  our  daily  lives  more  spatiality  implicit  in  what  is  ordinar- 

ily thought  of  as  symbolic  retrieval  than 
An  even  more  striking  instance  of  the  role  we  may  realize 
of  spatiality  in  common  experience  is  the 

coherence  and  persistence  of  "'imagery  '"  In  summary,  it  is  precisely  the  insight 

Two  individuals,  arguing  a topic  in  front  ot  about  how  we  tend  to  retrieve  items  from 

a blackboard,  will  refer  each  other  to  dia-  desk  tops,  from  files  and  bookshelves, 

grams,  equations,  and  terms  on  the  even  from  erased  blackboards,  that  lies 

basis  of  where  they  had  been  written.  at  the  heart  of  the  spatial  data  manage 

even  long  after  they  have  been  erased1  merit  concept  we  find  items  on  the  basis 

ot  a more  or  less  definite  sense  of  then 
location  m a familiar  space,  which  space 
may  be  actually  present  or  remembered 


This  well  evolved  human  ability  to 
organize  information  spatially  remains 
essentially  untapped  in  the  realm  of 
computer  based  information  handling 
1 vpically.  in  such  systems,  retrieval  on  a 
symbolic  or  name  basis  is  the  norm,  and 
must  tx'  the  norm  because  the  conven- 
tional keyboard  interlace  is  too  limited  a 
channel,  the  wrong  mode  and  medium, 
to  begin  to  otter  the  user  a direct,  palpa 
ble  sense  ot  spatiality 

In  the  spring  ot  1976.  the  Architecture 
Machine  Group  at  Ml  T proposed  to  the 
Cybernetics  Technology  Office  ot  the 
United  States  Defense  Advanced  Re 
search  Projects  Agency  (DARRA)  a pro- 
gram of  research  organized  under  the 
title  Augmentation  of  Human  Resources 
m Command  and  Control  through  Multi- 
ple Media  Man  Machine  Interaction 
Intrinsic  to  the  ensemble  of  studies  out- 
lined m the  proposal  was  a study  recall- 
ing the  ancient  principle  ot  using  spatial 
cueing  as  an  aid  to  performance  and 
memory  the  "Simonides  Effect 


Simonides  was  a poet  of  ancient  Greece 
famous  tor  his  ability  to  give  long  recita- 
tions entirely  from  memory  His  secret, 
which  as  a teacher  ot  rhetoric  fie  shared 
with  his  students,  w as  to  tie  each  sue 
cessive  part  of  a to-be-remembered 
poem  or  speech  to  a specific  locale 
within  the  mental  floor  plan  of  either  an 
actual  or  imagined  temple  Around  the 
floor  of  tfie  temple  were  statues,  serving 
to  augment  the  imagery  of  specific 
niches  or  corners 

To  commit  to  memory  a long  poem  or 
speech,  the  orator  would  pause  in  his 
imagination  before  the  threshold,  and 
w ith  this  mental  image  in  mind,  commit 
the  introductory  remarks  to  memory 
1 hen,  tor  each  successive  subsection  of 
the  talk  to  be  given,  the  orator  would 
mentally  walk  from  place  to  place  within 
the  temple  rehearsing  (lie  appropriate 
material  before  some  specific  piece  of 
statuary  Theresultai  ath  about  the 
temple  would  then  serve  as  a mental 
schema  to  organize  the  speech  upon  its 
eventual  recitation,  the  speech  being 
retrieved  and  reconstructed  during  ora- 
tion by  way  of  an  imaginary  tour  from 
statue  to  statue  around  tl  ic  temple  floor 


! 


Another  study  outlined  in  that  same  pro- 
posal looked  to  the  exploitation  of  virtual 
spatiality  as  a matrix  for  organizing  infor- 
mation the  "virtual  bulletin  board  " This 
term  refers  to  the  formulation  of  an 
implicit  space  which  occupies  no  "real" 
space  but  can  nonetheless  contain  a 
great  deal,  as  well  as  offer  the  same 
spatial  organizational  cueing  as  to  the 
whereabouts  of  material  that  real  work 
surfaces  do  The  tacit  size  of  a "field"  of 
information  to  be  portrayed  in  computer 
graphics  need  not  be  limited  by  the  size 
of  the  physical  display  Since  it  is  virtual, 
its  apparent  size  and  graphic  organiza- 
tion are  open  to  user  definition  in  the 
unconstrained  sense  of  Ivan  Sutherland's 
characterization  of  a computer  display 
as  "a  window  on  Alice  s Wonderland" 
("Computer  Displays."  Scientific  Ameri- 
can. June  1970) 


These  two  spatial  themes  specifically 
resonated  with  Dr  Craig  Fields.  Senior 
Program  Manager  in  the  Cybernetics 
Technology  Office.  As  a result  we  started 
to  explore  the  potentialities  of  a spatially 
oriented  data  management  system,  "a 
multidimensional  window  into  data " The 
outcome  has  been  that  over  the  past  two 
years,  under  the  aegis  of  DARPA.  the 
Architecture  Machine  Group  lias  been 
experimenting  with  the  creation  of  an 
information  management  system  whose 
distinguishing  characteristic  is  that  it 
exploits  the  user's  sense  of  spatiality  for 
purposes  of  organizing  and  retrieving 
data  a spatial  data-management 
system,  or  SDMS 
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This  page  The  monitor  to 
the  user  s right  as  seen  over 
the  arm  of  the  user’s  chair 
Fingertips  of  user's  hand  are 
on  the  small  joystick  On  the 
chair  s arm.  immediately  in 
front  of  user  s hand  is  a small 
touch- sensitive  pad  ready  for 
capture  of  fingertip  gesture 
input  The  other  arm  of  the 
chair  is  similarly  instrumented 

Opposite  page  The  absence 
of  any  desk-like  appur- 
tenances situated  in  front  of 
the  user  lends  a tone  of  imme- 
diacy and  directness  to  inter- 
action with  data  At  the  upper 
nght  comer  of  the  large 
screen,  one  of  the  sound 
system's  speakers  is  clearly 
visible 

The  lapboard  and  stylus  are 
not  featured  in  these  illustra- 
tions. For  a view  of  them,  see 
illustration  at  top  of  page  48 


The  Setting 


The  setting  ot  our  experimental  version  of 
SDMS  is  a multiple  media  information 
place  It  can  be  construed  as  an  image 
of  an  office  of  the  future 

The  physical  aspects  of  our  media 
room  are  easily  described  The  media 
room  is  about  the  size  of  a personal 
office  The  focal  point  of  the  room  is  a 
wall-sized  display  screen  Directly  in 
front  of  this  large  screen  is  an  instru- 
mented Eames  chair  On  either  arm  of 
the  chair  is  a small,  pressure-sensitive 
joystick  Adjacent  to  the  joystick  on 
each  arm  is  a two-inch-square  touch- 
sensitive  pad  Also  associated  with  the 
chair  as  complementary  instrumentation 
is  a ten-inch-square  data  tablet,  with  sty- 
lus. the  tablet  is  upholstered  to  serve  as  a 
lap  pad  This  tablet  bears  added  instru- 
mentation a small  microphone  mounted 
at  its  top  edge  to  allow  voice  input 

Two  small  television  monitors  are  situated 
one  on  either  side  of  the  chair  Each  of 
these  monitors  is  touch-sensitive  so  that 
the  user  can  point  to  data  or  can  input 
gestures  indicating  actions  to  be  taken 


Finally,  eight  loudspeakers  provide 
sound  output  for  the  media  room  Four  of 
them  are  placed  around  the  perimeter  of 
the  large-format  display  to  the  user's 
front  The  remaining  four  are  correspon- 
dingly arrayed  to  the  user's  rear  Octa- 
phomc  sounds  can  be  presented  to 
the  user  as  coming  from  various  appar- 
ent directions  in  the  space  of  the  room,  in 
coordination  with  actions  on  the  visual 
display  The  rate  of  playback  is  variable 
so  that  dynamic  sound/motion  effects, 
such  as  Doppler  shifts,  may  be  achieved 
We  call  these  "sound-synch  graphics 

Notice  that  there  is  a conspicuous,  but 
not  mandatory,  absence  of  a keyboard 

The  configuration  described  is  one 
which  evolved  through  previous  work  for 
the  Office  of  Naval  Research  (Contract 
number  0014-75-C-0460)  and  over  the 
course  of  our  experimentation,  but  this 
precise  ensemble  is  certainly  not  the  only 
one  that  might  have  been  adopted  The 
kernel  notion  of  managing  data  spatially 
is  not  necessarily  tied  to  a room-sized 
terminal  into  which  a user  goes  versus  a 
desk-top  arrangement  in  front  of  which 
he  or  she  sits. 


10 


A television  projector  in  the 
room  ad|acent  to  the  media 
room  back-projects  upon  the 
large  screen  to  the  user's 
front. 


The  relaxed  ambiance  conveyed  by  the 
presence  of  a chair  of  the  I ames  pome 
in  lieu  of  something  more  utilitarian  is 
intentional,  however  It  reflects  eonvie 
tions  and  positions  about  the  natuie  and 
tout1  of  human  comfuilt'i  inlomchon  that 
wt'  have  attempted  to  actualize  m the 
media  room  setting  Just  as  the  hands 
on  immediacy  ot  touch  sensitive  pacts 
suggests  a litoral  impatience  with  intan 
gibles  about  data,  so  the  decoi  as  efiito 
1111/ed  in  ttie  selection  ot  the  style  of  chan 
rebuts  the  premise  that  system  useis 
must  live  in  severe,  ascetic  settings 
(Set'  ttie  Prologue  to  tins  report ) 


We  have  attempted  to  eieate  an  mteitace 
wtucli  is  not  a tiny,  nanow  band  "port 
hole"  into  an  intomiation  bank,  that  bank 
itsell  an  abstractly  addressed  set  of 
intangibles  Rathei,  wt'  liave  attempted 
ratJic.rlly  forecast  ttie  setting  as  an  "mtor 
mational  surround'"  wtierein  the  user  is 
dnectly  engaged  witti  data  bodied  tenth 
in  vision,  sound,  and  touch,  data  inhabit 
mg  a spatially  definite  "vntual"  world  tliat 
can  be  inteiactively  explored  and  navi- 
gated 
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Finding  Data 


Spatial  data-rnanagement  involves  the 
creation  ot  a plausible  and  commodious 
' virtual''  space  at  the  computer  interface, 
together  with  a way  of  getting  around  in 
that  space  quickly  and  easily. 

The  spatial  world  of  SDMS  consists  of  a 
single  plane.  This  surface,  called  “Data- 
land,"  is  presented  to  the  user  in  two 
aspects: 

1 

it  is  continuously  visible  to  the  user  in  its 
entirety  in  an  "aerial."  top-on  view  dis- 
played on  one  of  the  monitors,  arbitrarily 
situated  to  the  user's  right,  called  the 
"world  view"  monitor:  and 


The  logical  relationship  between  these 
two  views  is  akin  to  a mapping  key.  which 
places  a particular  image  within  a larger 
locale.  The  large  screen  effectively  func- 
tions as  a "window"  and.  as  we  shall  see 
later,  a "magnifying  glass"  onto  Dataland. 

The  "world  view"  monitor  serves  specifi- 
cally as  a navigational  aid  to  the  user  in 
getting  around  Dataland.  The  large  dis- 
play of  whatever  portion  of  Dataland  is  so 
"close  up"  that  the  user  would  get  lost 
easily  if  there  were  not  always  on  view  a 
map  of  the  entire  Dataland  world.  A 
small,  highlighted,  "you-are-here”  rec- 
tangle on  that  display  shows  the  user  at 
all  times  the  position  of  the  large-screen 
window  on  Dataland. 


Dataland  as  It  appears  on  the 
world  view”  monitor  to  the 
user's  right.  Notice  highlighted 
you-are-here”  rectangle  sit- 
uated over  a map  of  north- 
eastern United  States  located 
in  upper-left  region  of  Data- 
land. "You-are-here”  marker 
is  translucent,  under  joystick 
control,  and  indicates  the 
region  of  Dataland  visible  on 
the  large  screen  to  user's 
front. 


2 

simultaneously,  a small  subsector  of  the 
Dataland  surface  is  displayed  on  the  ten- 
foot  diagonal  screen  to  the  user's  front, 
vastly  enlarged  and  with  appreciable 
gain  in  detail. 


To  illustrate  how  an  actual  Dataland 
might  appear  to  a user  seated  in  the 
media  room's  control  chair,  let  us  assume 
a database  already  created.  In  practice, 
individuals  would  design  and  structure 
their  own  Datalands  according  to  per- 
sonal preferences,  tasks,  and  needs,  just 
as  the  contents  and  organization  of 
anyone's  desk  top  come  to  reflect  some 
combination  of  job  requirements  and 
individual  taste.  13 


Illustrations  on  this  page  and 
the  next  depict  relationship 
betweeen  world  view  mon- 
itor and  the  large  screen.  On 
Dataland  view,  this  page,  the 
translucent  you-are-here 
marker  resides  over  lower- 
left-most  item.  The  item,  a 
calendar,  appears  on  the 
large  screen,  much  magni- 
fied and  with  finer  detail 
(opposite  page). 


Now,  what  is  visible  to  the  user  on  the 
surface  of  this  example  of  Dataland? 

Items  of  various  sorts  may  be  seen 
Some  are  small  photographs  pictures  of 
animals,  of  people,  even  a miniature 
LANDSAT  satellite  ptioto  of  part  of  New 
England.  Other  items  look  like  small 
maps.  We  see  what  appear  to  be  book 
covers,  television  sets,  a business  letter, 
even  a small  image  of  an  electronic 
hand-held  calculator  Yet  other  items 
resemble  emblems  or  "glyphs,"  not  dis- 
similar from  business  logos. 


How  does  one  get  around  in  such  an 
informational  landscape? 

User  control  of  navigation  around  Data- 
land is  provided  by  "|oystick"  action  The 
small  joystick  in  the  right  arm  of  tfie  chair 
permits  a helicopter-like  flight  over  the 
surface  The  joystick  itself  does  not 
deflect,  but  responds  to  pressure.  It  is 
"self-centering"  in  that  it  registers  zero 
deflection  when  not  touched  Push  the 
stick  “away"  from  you  and  the  helicop- 
tering motion  over  Dataland  is  north,  or 
toward  the  top  of  the  screen.  Pull  the  joy- 
stick toward  you  and  motion  is  south 
Pressure  to  the  right  and  left  will  direct 
you  east  or  west 
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The  apparent  rate  of  travel  is  proportional 
to  the  pressure  the  user  exerts  on  the  joy- 
stick At  maximum  pressure,  it  takes 
about  twenty  seconds  to  travel  across 
Dataland,  as  illustrated  in  this  example 
database,  from  its  eastern  to  western 
borders 

During  flight,  the  user  is  given  visual 
feedback  of  two  kinds 

1 

movement  of  the  material  across  the 
"window''  of  the  large  screen; 

2 

tracking  of  the  path  of  the  window  by  the 
highlighted  rectangle  on  the  auxiliary 
monitor 


Let  us  say  that  you  push  the  joystick  to 
the  right  and  away  from  you  This  action 
will  set  you  helicoptering  across  Data- 
land to  the  "northeast."  On  the  big  screen 
you  will  see  material  scudding  by  in  a 
southwesterly  direction;  that  is.  the  intui- 
tive sense  of  movement  will  be  that  of  a 
window  looking  out  over  a world  passing 
by  below  Simultaneously,  the  highlighted 
rectangle  on  (tie  auxiliary  view  at  your 
right  hand  will  track  your  northeastward 
progress. 

Complementing  joystick  control  of  flight 
over  Dataland  is  "teleporation"  by  direct 
touch  Traversing  relatively  large  dis- 
tances with  the  joystick  can  be  frustrating; 
tor  the  veteran  user,  this  often  represents 
"dead  time."  Accordingly,  the  user  is 
given  the  option  ot  indicating  where  tie 
wants  to  go  directly  on  the  touch- 
sensitive  "world  view " The  highlighted 
"you-are-here"  rectangle  will  disappear 
from  its  current  location,  to  reappear 
centered  at  the  sjiot  indicated  by  the 
finger  touch.  On  the  large  screen,  the 
"magnified"  view  of  the  old  location  dis- 
appears, to  be  replaced  by  the  |ust 
indicated  locale. 


While  its  specific  format  is  much  a per- 
sonal matter,  the  graphic  layout  of  Data- 
land  can  aid  navigation  significantly. 

First,  the  directness  and  simplicity  of 
Dataland  itself  provides  a unitary  con- 
ceptual substrate:  a large  plane  con- 
jointly given  via  two  correlated  views  on 
the  small  and  large  screens.  Second,  the 
absolute  and  relative  location  of  objects 
and  items  is  evident  within  the  frame  of 
the  Dataland  world.  Absolute  locational 
cues  are  provided  by  the  main  "frame''  of 
Dataland:  the  fact  of  edges,  corners,  up- 
down  and  left-right  directionality.  Any 
item  is  perceived  and  remembered  in  the 
context  of  and  juxtaposed  with  certain 
other  items.  Third,  the  use  of  local  back- 
ground colors  affords  a further  visual 
sense  of  neighborhood,  grouping  certain 
items  together  Color  backgrounding 
can,  of  course,  be  "nested,"  color  back- 
ground upon  color  background,  as  it  is  in 
our  "rogues'  gallery"  of  Architecture 
Machine  Group  personnel  in  the  south- 
east corner  of  Dataland  Within  the  over- 
all rectangular  area  there  is  further  group 
intraorganization  indicated  by  back- 
ground colors 


Teleportation"  about  Data- 
land by  direct  touch:  At  top. 
user  touch  indicates  item 
within  a group  with  orange 
background  as  what  he 
wants  to  go  to  on  Dataland. 
This  item,  a "book  cover", 
fills  screen  to  user's  front  as 
user  next  indicates  a location 
to  the  right  and  somewhat 
lower  (middle).  "You-are- 
here"  marker  immediately 
moves  to  this  new  locale, 
and  large  screen  view  now 
bears  magnified  view  of  the 
newly  selected  item,  a sche- 
matic floor  plan  of  a 
museum. 


I 


17 


Faces  which  are  unrecogniz- 
able and  tiny  on  Dataland  as 
seen  on  the  world  view" 
monitor  become  detailed  and 
clear  when  zoomed-m  upon 
on  the  large  screen. 


The  level  of  visual  detail  in  items  pro- 
vided on  the  auxiliary  world-map  view 
deserves  comment  In  everyday  life,  we 
need  some  minimal  amount  of  visual 
detail  to  identify  an  item  that  we  have 
never  seen  before  If  an  item  is  small  in 
size,  or  small  because  distant,  it  can  lack 
sufficient  shape  or  figural  detail  to  enable 
us  to  discern  what  it  is.  However,  recog- 
nition of  already  familiar  items  requires 
less  figural  detail.  We  readily  recognize 
friends  way  down  a street,  at  distances 
where  visual  acuity  is  insufficient  to  pro- 
vide facial  details.  We  recognize  them  on 
the  basis  of  gait,  relative  size,  general 
shape  or  "visual  envelope,"  as  well  as  by 
context,  that  is.  by  expecting  a figure 
with  such-and-such-color  clothes  to 
appear  in  such-and-such  a place. 


Or.  take  the  example  of  a Coca-Cola  sign 
down  the  street.  At  a distance  at  which  we 
cannot  distinguish  the  individual  letters 
of  the  sign,  we  yet  recognize  it  as  being  a 
"Coke"  sign.  The  fact  that  it  appears  over 
a drugstore  or  variety  store  supports 
contextually  this  act  of  recognition. 
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A similar  principle  holds  in  the  area  of 
audition.  On  a certain  popular  television 
game  show  ("Name  That  Tune"),  per- 
sons who  are  particularly  well  informed 
about  popular  music  as  well  as  alert  can 
sometimes  identify  ("recognize")  a song 
on  the  basis  of  no  more  than  its  first  two 
or  three  notes.  Consider,  for  example,  the 
first  two  notes  of  the  melody  "Auld  Lang 
Syne."  The  clue  that  the  song  about  to  be 
heard  is  associated  with  a holiday,  the 
distinctive  tonal  interval  of  the  first  two 
notes  of  the  song,  can  be  sufficient  to 
define  the  intersection  of  context  (New 
Year's  Eve)  and  tonalities,  as.  indeed. 
"Auld  Lang  Syne." 

The  recognition  situation,  as  opposed  to 
the  identification  situation,  is  what  we 
anticipate  to  be  the  operating  condition 
for  the  typical  user  of  SDMS.  That  is,  the 
array  of  highly  familiar  items  in  Dataland 
is  "searched."  not  by  a deep  visual  anal- 
ysis of  every  item  before  one  s view,  but 
by  relatively  "shallow"  visual  processing, 
quite  adequate  for  the  recognition  of 
familiar  items  in  familiar  places. 


MM* 
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If  a user  has  been  working  with  a certain 
personal  Dataland  configuration  for 
some  time,  it  is  reasonable  that  all  that 
should  be  needed  on  the  auxiliary  view 
of  Dataland  is  sufficient  detail  to  recog- 
nize items  readily  on  that  surface  With 
constant  usage  certain  items  could 
undergo  further  reduction  in  visual  detail, 
even  to  the  extent  of  becoming  simple 
colored  squares,  circles,  or  triangles. 
This  extreme  represents  a “cueing"  situ- 
ation, rather  than  a recognition  situation 

Qualitatively,  the  figural  resolution  of 
items  in  Dataland  as  they  appear  on  the 
world-map  monitor  fall  in  the  intermedi- 
ate range  on  the  spectrum  cueing,  rec- 
ognition. identification.  That  is.  they  serve 
readily  for  recognition  by  an  accustomed 
user,  but  not  necessarily  for  identification 
upon  a first-time  look  by  a new  observer. 


Perceptibility  indeed  differs  somewhat 
from  item  to  item  For  example,  the 
LANDSAT  satellite  image  does  look  like 
some  bit  of  geography,  but  it  is  not  clear 
whether,  from  a farthest-out  view,  it  is  a 
LANDSAT  view,  a map.  or  a drawing. 
The  user  who  knows  what  it  actually  is  of 
course  readily  identifies  it  and  refers  to  it 
as  "the  LANDSAT  photo ."  Or.  consider 
the  set  of  photographs  in  the  southeast 
corner  of  Dataland.  They  are  sponta- 
neously classifiable  as  photos,  but  even 
for  those  wlio  are  personal  acquaint- 
ances of  the  people  pictured,  the  photos 
are  ever  so  slightly  too  small  and  unde- 
tailed to  permit  positive  identification 
without  zooming  in.  Nonetheless,  it  is 
clear  from  "afar"  to  any  observer  that  the 
items  are  in  fact  photos  of  faces. 


Sound,  too.  can  be  an  invaluable  naviga- 
tional aid  In  SDMS.  space  as  cued  by 
sound  can  have  an  implicit  extent  much 
greater  than  that  bounded  by  the  visual 
"window1 " onto  Dataland  given  by  the 
large-screen  view  Off-screen  voices, 
becoming  gradually  audible  as  one 
approaches  some  item  or  area,  can 
serve  to  orient  and  guide 

Two  examples  of  sounds  that  we  have 
employed  include: 

1 

an  off-screen  voice,  in  stereo,  reciting 
"This  is  MIT  This  is  MIT  the  voice 
leading  the  traveler  in  Dataland  first  to 
the  vicinity  of.  and  then  to.  a certain 
"glyph"  bearing  the  initials  MIT.  winch 
was  the  source  of  the  sound: 

2 

a gray  telephone  with  white  touch-tone 
buttons  (not  activated  for  "dialing."  but 
potentially  so),  which  rings  louder  and 
louder  upon  approach 


These  and  other  sounds  did  much  to 
create  a sense  of  "region"  and  " neigh- 
borhood." a feeling  of  proximity,  much  as 
the  bells,  hammerings,  traffic  noise,  and 
playground  chatter  of  tfie  real  world  fill 
things  out  At  moments  in  navigating 
across  Dataland.  depending  upon  the 
route  taken,  sound  can  be  the  sole  direc- 
tional cue.  should  one  momentarily  not 
be  consulting  the  "world  view"  auxiliary 
monitor  but  watching  the  large  screen 

We  note  in  passing  that  tire  user  is  given 
interactive  control  of  the  loudness  level  of 
sound  through  gesturing  upon  either  of 
the  small,  touch-sensitive  pads  on  the 
arms  of  the  chair.  Circling  with  the  finger- 
tip in  a counterclockwise  direction  atten- 
uates the  volume  of  sound,  and  a circling 
gesture  clockwise  causes  volume  to 
increase. 

With  respect  to  navigation  around  Data- 
land. we  have  so  far  talked  only  about 
movement  over  the  surface  of  the  plane 
of  data  at  a constant  "altitude."  How  then 
does  the  user  manage  to  get  at  the  data 
detected  at  a distance? 


The  Northeastern  Coast  ot 
the  United  States  seen  trom 
atar.  A satellite  Image  of  the 
New  England  coastline  is 
Inset  at  the  edge. 
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Zooming-in  upon  Dataland  s 
map  of  Southeast  Asia:  suc- 
cessive images  on  this  page 
depict  a monochromatic  map 
becoming  larger  and  block- 
ler  in  texture  The  image 
then  gains  both  clarity  and 
color  (opposite,  top)  for 
scrolling  about  (opposite, 
middle,  and  bottom). 


While  the  right-hand  |oystick  guides  lat- 
eral and  up-down  movements,  the  joy- 
stick mounted  in  the  left  arm  ot  the  media 
room's  control  chair  permits  the  user  to 
zoom'  in  upon  Dataland  for  a closer 
look  Pushing  forward  on  the  left-hand 
joystick  wi'i  cause  zooming  in.  pulling 
back  will  cause  zooming  out  Upon 
zooming  in.  the  image  of  the  item  on  the 
large  screen  gains  in  size  dramatically 
The  relative  size  of  the  "you-are-here 
highlighted  rectangle  on  the  "world  view 
auxiliary  monitor  grows  correspondingly 
smaller,  yet  remains  on  view  so  that  the 
user  will  always  know  his  location  in  the 
spatial  context  ot  Dataland 


For  example,  on  the  world  view  auxil- 
iary map.  we  can  recognize  a small, 
monochromatic  map  of  Southeast  Asia 
located  in  the  western  part  ot  Dataland. 
at  about  mid-latitude  Coming  in  closer 
upon  the  map  by  pressing  forward  on 
the  left-hand  joystick  makes  detail  more 
discernible,  as  well  as  causing  the  car- 
tography to  gam  color 
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Notice  that  the  enlargement  of  the  map 
image  takes  place  in  "stages " That  is. 
the  image  upon  zooming  in  becomes 
larger  in  discrete  increments,  simultane- 
ously more  "blocky”  in  appearance  At  a 
certain  point  in  zooming  in.  the  blocky 
image  is  replaced  by  a clear  map  image 
the  same  size  and  in  the  same  register 
as  the  previous  blocky  one 

The  increasing  "blockiness"  of  the  image 
accompanying  the  increase  in  size  dur- 
ing zooming  in.  requiring  eventual 
replacement  with  a clearer  image,  is  a 
consequence  of  the  digital  nature  of  the 
television  image  and  of  the  way  the 
zoom'"  effect  is  achieved 





In  broadcast  television,  the  "information''  The  digital  color  image  in  SDMS  can  be 
for  the  image,  namely  the  brightness  conceded  of  logically  as  composed  of 

(luminance)  and  color  (chrominance)  an  array  of  tiny  dots  called  "picture 
information  for  each  scan  line  compos-  elements,  or  "pixels  Each  pixel  corre- 
mg  it.  comes  over  the  airwaves  as  an  sponds  to  certain  numeric  values  locat- 

analogue  signal  With  digital  television  ed  in  addressable  slots  In  the  frame 
however,  the  information  which  modu-  buffer  Given  this  mosaic,  a ready  way  to 

lates  the  image  either  is  generated  by  blow  up  the  image  for  a zoom-like  effect 

computer  computations  or  comes  is  to  repeat  (in  the  sense  of  displaying 

directly  out  of  the  computer's  memory  again")  each  picture  element,  in  both  the 
store  The  color  television  images  dis-  horizontal  and  vertical  dimensions  of  the 
played  in  SDMS.  with  important  excep-  image,  first  once,  then  twice,  then  three 

tions  to  be  noted  later,  actually  reside  as  times,  and  so  forth.  The  repetition  of 

stored  information  in  an  area  of  computer  pixels  is  patterned  from  center  screen 

memory  termed  a "frame  buffer"  outward  in  emulation  of  a central-axis 

"optical"  zoom.  The  result  visually  is  a 
negatively  accelerating  rate  of  image 
expansion. 

The  basic  Dataland  image,  as  seen  from 
the  highest  "ceiling"  or  altitude,  actually 
incorporates  one  "repeat"  to  begin  with, 
partly  to  avoid  the  rather  large  visual 
"lurch"  inherent  in  the  quadrupling  of 
image  size  that  occurs  when  each  pixel 
is  first  repeated  once  vertically  and  hori- 
zontally. One  initial  "repeat"  makes  for  a 
smoother  total  effect  even  though  only 
half  of  the  television  screen's  real  resolu- 
tion is  being  used 
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The  increasing  "blockiness'  of  the  image 
upon  zooming  in  derives  precisely  from 
the  squarish  pattern  of  repetitions  of  the 
individual  pixels,  the  basic  pixel  being 
visually  just  a small  dot  of  color.  To  coun- 
teract this  side  effect  of  the  way  we 
achieve  zoom  effects,  the  now  larger  and 
biockier  image  can  be  replaced  by  an 
image  of  the  same  size  and  in  the  same 
register  but  with  increased  detail  and 
clarity. 

In  practice,  the  zoom  procedure  we  have 
adopted  calls  for  the  replacement  of  the 
scaled-up.  “blocky"  image  at  about  a 
repeat  count  of  seven,  so  that  the  loss  in 
clarity  with  increasing  blockiness  is  not 
too  great.  With  successive  rounds  of 
repeat  count  incrementing  followed  by 
image  replacements,  a zooming  in  proc- 
ess could  be  as  “magnifying''  as  one 
wished,  the  practical  limitation  being  the 
resolution  of  the  source  image  information. 

Other  techniques  of  zooming  are  under 
investigation,  in  particular  interpolate 
zoom,  which  removes  blockiness  and 
jumpiness  but  manifests  the  same  deg- 
radation in  resolution. 


On  the  large  screen  to  the 
user's  front,  image  of  William 
Kelley  of  our  laboratory's  staff, 
at  first  small  (opposite  page), 
becomes  larger  and  "biock- 
ier" as  the  user  zooms-in  (top 
and  middle,  this  page).  At  a 
certain  point  in  approach,  the 
blocky  image  is  replaced 
(instantaneously)  with  a 
higner  resolution  image 
(bottom). 
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Al  a distance,  and  at  low 
resolution,  this  image  on 
Dataland  is  clearly  a letter 
Letterhead  logos,  despite 
the  coarseness  of  scale, 
generally  can  serve  as 
reliable  cues  for  sender  s 
identity. 


While  it  might  at  first  seem  that  such  dis- 
continuities in  image  sharpness  during 
zoom  can  only  be  disadvantageous, 
there  are  circumstances  in  which  a lack 
of  low-order  visual  detail  followed  try  the 
sudden  introduction  of  finer  detail  makes 
good  sense  in  terms  of  information  pre- 
sentation 

In  the  upper  right,  or  northeast,  comer  of 
Dataland  is  an  area  allocated  to  Architec 
lure  Machine  Group  "correspondence" 
with  our  research  sponsor,  DARPA 
In  this  area  is  a facsimile  of  a letter,  com- 
plete with  letterhead  and  signature. 
When  approached,  the  letter  indeed 
looks  like  a letter  but  is  ns  yet  unmade 
trie  At  a certain  threshold  nearness,  the 
text  suddenly  gains  in  clarity  and 
becomes  perfectly  legible. 


To  zoom  in  upon  an  item  of  interest  is  to 
'address"  that  item  for  purposes  of 
inspection  For  some  items,  simply  to 
come  up  lo  them  will  cause  them  to  gam 
not  only  in  size  but  in  information: 
increased  detail,  finer  delineation,  color, 
anil,  in  some  cases,  sound  and  move- 
ment To  approach  them  is  ter  “activate" 
them.  With  other  items,  some  further  user 
initiative  such  as  "touch  selection"  may 
be  involved  as  a part  ot  a more  interac- 
tive process  of  perusal. 
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Perusing  Data 


Logically  distinct  fiom  finding  a certain 
item  in  Dataland  is  perusing  that  item  in 
place  Once  the  user  has  found  the  item 
and  zoomed  in  on  it.  (tie  specific  foim 
that  perusal  takes  is  conditioned  by  the 
nature  of  the  data 

In  direct  conti ast  to  material  that  is  "out 
there."  visible,  to  be  developed  in  lull 
simply  by  approach,  aie  materials  which 
need  to  be  "perused,”  where  perusal 
means  opening  up  a volume,  starting  a 
film  clip,  selecting  part  ol  a set  of  slides 
to  ire  seen,  and  so  on  I he  essential  idea 
is  that  the  user  opens  up  and  develops 
his  interchange  with  the  data  more  or 
less  interactively,  thiough  direct  touch  in 
a "hands-on"  manner  The  dictionary 
sense'  of  "perusal"  is  that  of  "survey  01 
scrutiny,  in  some  thoroughness  or  detail " 
It  is  this  engaged  quality  of  person  data 
interchange  that  we  intend  fry  this 
terminology 


Our  notion  ot  pemsal  includes  a class  of 
"data  types"  which,  when  addressed, 
prove  to  bo  unusually  rich  in  motion, 
colot,  and  sound  One  ot  these  new  data 
types  is  yet  an  olei  and  lamiliai  one 
the'  "book  " 

1 lit'  user  zooms  in  on  what  appears  Item 
atai  lei  be  simply  a rectangle  At  close 
enough  range,  the  coloi  patch  gams 
detail  in  the  loim  ot  some  horizontal 
black  lows  of  "characters"  implying  a 
punted  title'  Upon  e've'n  ye't  ne'.lie'i 
appieuch,  tlie'  black  tiguie'S  give  wav  to 
maeiablo  te'\t.  giving  the'  title'  eil  the'  ite'in. 
which  is  netw  clextrly  sevn  as  imply  mg  a 
"book  like'"  image' 

I lie'  usei,  ot  e'eiutSe'.  knows  that  the'  lev 
tangle'  is  a "txxik."  having  put  it  them  in 
the'  lust  place'  I leiWe'Ve'l,  it  tie'  is  a Slit' 
scribe'!  te'  a e'Ompute'i  iie'twoik  base'ei 
"bexik  ot  the'  month"  se'ivie'e.  aiiei  has 
pei haps  mgue'ste'ei  eir  has  a staiielmg 
Olele'i  tor  beX'ks.  the'  t'eX'k  may  lx'  eille'  lie' 
has  ye't  to  sex',  having  |ust  iippexueef  on 
the'  Dataland  suifaee  in  a ce'rtam  are'a  on 
the'  elata  plane  se'iving  as  a "Ixxik  dteip 
tor  incemimg  eximmunicatiems  which  an' 
txiok  like' 


Upon  nearest  approach,  when  the  image 
of  the  book  is  fully  addressed  in  the  vis 
nal  sense  of  tilling  the  large  screen,  a 
table  of  contents"  lor  the  book  appears 
on  the  second  and  so  tar  unused  color 
monitor  to  the  user's  left  T Ins  table-ot 
contents  view  is  one  ot  a class  of  interac 
five  key  maps"  made  available  to  the 
user  as  ad|imcts  to  certain  data  types,  to 
permit  direct  user  interaction  with  the 
data  to  be  perused 

In  this  instance,  the  key  map  allows  user 
access  to  the  interior  ot  the  "book  on 
display,  exactly  as  the  table  ot  contents 
ot  a "real"  book  lets  the  reader  (a)  know 
wtiat  the  book  otters  by  way  ot  topics 
covered,  and  (b)  serves  as  a "dispatch 
ing"  table  to  send  the  reader  directly  to 
the  subsection  of  the  book  that  is  ot  most 
immediate  interest 


User  interacts  by  touch  with 
a table  ot  contents  key 
map  tor  book  like  data  types 

In  close-up  ot  this  key  map 
below,  note  touchable  tabs 
aligned  with  heading  taglines 


Upon  touch  indicating  chap- 
ter heading  ot  SOUND  (top), 
the  heading  expands  (bot- 
tom) to  Oder  sub-headings 
under  SOUND 
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Notice  that  the  table  ot  contents  can 
'expand"  to  show  a liner  breakdown  ot 
content  area  when  the  user  selects  a 
major  hearting  ot  interest.  In  concert  with 
the  activity  on  the  key  map  monitor 
screen,  the  main  large-screen  view  will 
"go  to"  the  section  ot  the  book  selected 
by  the  user  through  the  interactive 
key  map  so  that  the  user  may  read  the 
material 

There  is  yet  further  interaction  with  the 
data  type  ot  book:  that  ot  turning  its 
pages  Tins  action  is  initiated  by  a spe 
cific  user  action:  a page  turning  gesture 
given  as  a right-to-left,  top  to- bottom 
stroke  on  either  ot  the  ten  ich  sensitive 
pads  on  the  arms  ot  tire  user  chair  Any 
page  can  be  turned  back  try  a stroke 
across  the  pad  in  the  opposite  direction 
The  accompanying  visual  action  is  the 
display  ot  an  "animation"  of  a page  actu 
ally  turning.  The  upper  right  hand  corner 
ot  the  page  on  view  progressively  "lifts" 
away  and  sweeps  leftwards  across  the 
screen,  immediately  revealing  the  new 
page  below 


ASCII  TEXT  DISPLAY 
This  is  a pa  tie  of  ASCII  text 
displayed  by  the  new  8S  font  system. 
The  font  master  is  stored  "off  screen" 
starting  at  page  0 of  the  extended 
memory.  At  the  ASCII  text  is  parsed, 
each  charactei  is  “moved",  via  the 
microcode,  to  the  appropriate  -pot  in 
the  car  lent  ly  visible  pace. 

The  fonts  ace  two  bits  pei  pixel. 
Associated  with  each  font  i a 
location  table  giving  it- 
cooordinates  on  the  invisible 
lead- image.  Also  with  the  font  is  a 
one  dimensional  Kerning  table.  The 
Kerning  table  is  stored  in  progiam 
memory  during  the  toi  mat  ting  phase 
of  display. 

The  page  is  flipped  by  scratching 


This  is 
displayed  by  the  new  8S  for^ 
The  font  master  is  stored 
start  inq  at  paqe  0 of  the  el 
memory.  As  the  ASCII  text] 
each  character  i "moved" 
micr ocode.  to  the  appr  >pri^ 
the  currently  vi  ible  ;pac 
The  fonts  are  two  bits 
Associated  with  each  font 
locat ion  table  giving  it 
owrdmates  >n  t he  invi  ibl 
read-  image.  A l with  the | 
ne  dimensional  Kerning  tab 
Kerning  table  is  stored  in 
memory  during  the  for  mat  tin 
of  display. 

The  page  is  flipped  by  s 


eac 


appropr  late 
ve 

consist 


t he  c ur  r « 

The  font 

Associated  •*»» 
locat ion  t 
. ooor di nates 
r ead- image . 

' ne  d i men  i 
Ker  n mg  t abl< 
me  m.  ■ r y dur  l n 
f display. 


direction  on  a 
)oy-pad.  A "page" 
s of  two  paqes  of 
the  low 
per  pixel 

ot  her 

bits.  By 
display 
page  to 

a template 
four  bit  : jf 
■odit  y inq  t he 
.hove, 
template  ate 
back  and 
As  t he  paqe 


pane  is 


Initial  page  ot  text  (left,  top) 
begins  to  "turn'1  (lett.  middle) 
upon  user  indication  (a 
stroke  ot  the  linger)  on 
touch-sensitive  pad  on  arm 
of  user  chair. 

The  nest  page  ot  text  is 
progressively  revealed  as 
the  ' page  edge  sweeps 
leftwards,  and  is  almost 
entirely  on  view  near  the  end 
ot  the  page  animation 
sequence  (right,  bottom) 


start inq 
memory.  As 
each  character  is 
microcode,  to  the 
the  cur  tent  ly  vis 
The  fonts  are  t 
Assoc l at ed  with 
location  table  qi 
cooordinates  on  the 
reed- image, 
one  dimensional 
Xeininq  table  is 
eemory  dur  inq  t he 
of  display. 

Tha  paq#  is  fit 


the  appropriate  direction  on  a 
t ouch- sans  it  ive  joy-pad.  A *paqa" 
ctually  consists  of  two  paqas  of 
haracters,  one  occupying  the  low 
|dar  two  bits  (of  the  8 bit  pet  pixel 
£plsy  image'  and  the  other 
pymq  the  next  two  bits.  By 
inq  the  color  matrix,  display 
(switched  from  one  paqe  to 

Shadow-,  are  a template 
hiqh-ordet  four  bit  of 
.qe.  By  modifying  the 
change  ment loned  above, 
its  ot  the  template  are 
the  had--  ba*K  and 
[the  page.  As  the  page 
e being  exposed  is 
jh  the  shadows, 
the  current  page  is 


£ 
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The  image  of  the  television 
set  serves  as  locale  marker 
for  television  data  types  on 
Dataland  On  the  tube  face  of 
the  set  can  be  a program 
guide'  . or.  as  in  the  example 
on  the  following  pages,  a pic- 
torial label  which  becomes 
dynamic  upon  approach 


me  intent  is  not  to  give  a nostalgic 
impression  ot  the  way  hooks  usocl  to 
look,  like  the  electric  fireplace  with  plastic 
hack  lit  logs  The  purpose  is  quite  tunc 
tional  First,  the  page  is  a unit  of  appre 
hension  of  the  hook,  it  is  a "trite  sicod" 
chunk  or  packet  ot  the  material  that  one 
teels  one  can  take  in  comfortably,  in 
contrast  to  the  stretching  ever  onward, 
rhythmless  character  of  scrolled  text 
Second  tht'  page  serve's  as  a progress 
marker  through  the  text  in  a " milepost 
sense  With  continuous  scrolling,  such 
natural,  discrete  units  of  progioss  are  not 
explicit  m the  medium,  and.  it  felt  neces 
sary  try  the  reader,  must  he  ettor ttully 
abstracted,  resulting  in  a more  burden 
some  tone  to  the  leading  task 


1 hus.  the  page  flipper  is  not  Involous  It 
ptov ides  the  user  with  feedback  that  a 
page  has  turned,  the  directionality  ot  the 
animation  corresponding  to  forward  or 
backward  Specifically,  it  reinstates  tor 
the  user,  in  the  context  ot  the  reading  ot 
computet  television  text,  the  functional 
eguivalent  ot  what  the  fact  of  the  page 
gives  to  the  loader  ot  traditional  paged 
punted  media  the  sense  of  knoyymg 
whore  you  are  in  the  material  Mon'  gen 
orally,  it  is  an  instance  ot  what  y\o  choose 
to  term  a "media  fiducial,"  a marking 
system,  ot  built-in  feature  ot  any  medium 
that  serves  the  function  of  lotting  you 
know  "whoio  you  are"  based  on  looa 
tional  cues  arising  out  ot  the  basic  struc 
tine  ot  the  medium  itself 


One  tadical  departure  tiom  the  otdi 
nary  concept  ot  the  hook,  except  as 
adumbrated  in  such  page  cornet  flip 
animations  as  this  report  has  in  its  uppoi 
left  hand  page  comets,  is  the  tact  that 
while1  some  page  illustrations  will  he  still 
slides,  other  illustrations  will  ho  mox  ios." 
or  animated  diagiams.  with  01  without 
sound  accompaniments  The  possibility 
ot  sound  can.  upon  occasion,  permit  the 
l look  to  comment  upon  its  own  text  and 
illustrations,  and  even  read  itself  to  you 


Another  unusual  data  type  residing  in 
Dataland  is  that  of  television  An  instance 
of  this  is  the  presence  of  a "glyph"  on  the 
Dataland  surface  which  is  the  image  of  a 
Sony  television.  Zoomed  in  on.  the  televi- 
sion set  becomes  very  large,  its  own 
screen  filling  the  large  screen  directly 
before  the  user.  What  is  at  first  a rather 
sketchy  black-and-white  image  on  the 
tube  face  of  this  “virtual  television"  is 
then  replaced  with  live  action:  "Columbo." 
The  Sting. ' or  a documentary  on  how  to 
stop  smoking. 

What  may  be  seen  on  this  television  set 
in  Dataland  is  anything  that  can  be  put 
on  television,  live  or  recorded,  closed- 
circuit  or  broadcast.  The  user  could,  for 
example,  indicate  in  the  context  of  a ses- 
sion with  SDMS  that  he  next  wants  to  view 
the  twelve  o'clock  news,  or  see  a rerun 
of  some  footage  that  was  taped  earlier. 


a 


Zooming-in  approach  to 
television  set  on  Dataland 
(left  column),  enlarges  the 
television  set  s screen  to 
become  effectively  as  large 
as  the  screen  to  the  user  s 
front  (right  column)  The 
blocky  digital  image 
becomes  replaced  by  a 
dynamic  image  in  color 
from  videodisc,  launching 
into  continuous  live  action 
(opposite  page). 


Since  recorded  sequences  that  might  be 
viewed  on  the  television  are  processes 
taking  place  over  tune,  the  key  map  for 
interaction  with  sequences  on  the  virtual 
television  set  makes  available  to  the  user 
ways  of  controlling  events  that  are 
oriented  over  a time  span 

An  example  of  such  a key  map  is  shown 
in  the  accompanying  illustration  Central 
to  the  key  map  is  a •dial.''  whose  circum- 
ference indicates  the  time  span  to  be 
covered  by  [he  entire  movie  sequence, 
with  a pointer  hand  showing  the  current 
time  in  passage.  One  revolution  of  this 
semantic  clock  can  be  seconds,  min- 
utes. or  hours.  The  total  time-to-go  is  dis- 
played by  a digital  counter  on  the  key 
map;  minutes  and  seconds  elapsed  are 
also  displayed  digitally 


Sliding-bar  cursors  toward  the  bottom  of 
this  key  map  allow  the  user  to  speed  up 
or  slow  down  the  rate  at  which  material  is 
shown,  as  well  as  to  have  it  shown  in  a 
forward  or  backward  sequence  The 
material  can  be  simply  a "movie,'  in 
which  case  it  probably  makes  little  sense 
to  show  backwaid  slow  motion  But  if 
the  material  happens  to  be  an  instruc- 
tional movie,  for  example,  about  how  to 
tie  a sailing  knot,  or  do  a sleight-of-hand 
card  trick,  then  this  interactive  ability  to 
regulate  time  can  be  invaluable.  The 
material  to  be  shown  in  such  a format 
can  be.  in  general,  any  material  that  can 
be  viewed  visually  and  sequentially 
movies,  television,  animations,  slide 
sequences,  as  well  as  any  electronic 
product  of  television  mixing  and  special 
effects. 
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User  interaction  with  dial 
Key  map  user  sets  pace  to 
be  somewhat  slower  than 
normal  (top.  left)  Sequence 
•s  stopped  (freeze  frame)  by 
putting  finger  on  pointer  of 
dial  (middle),  and  lesumes 
when  finger  is  lifted  (bottom, 
right). 


The  visual  repertoire  of  our  virtual  televi- 
P7820  sion  screen,  as  well  as  other  -slide  col- 

lections" about  Dataland.  is  markedly 
augmented  by  optical  videodisc  backup 
E ther  side  of  this  laser-scanned  disk  can 
hold  up  to  54,000  individual  frames  Any 
frame  can  be  individually  addressed 
(freeze  frame)  or  arbitrarily  selected  (ran- 
dom access),  or  sequences  of  frames 
can  be  exhibited  forward  or  backward  at 
varying  rates.  The  optical  videodisk  as 
storage  vehicle  has  a dramatic  cost 
impact  on  SDMS.  inasmuch  as  the 
player  costs  less  than  $2500 

In  a more  general  sense,  the  television 
"set"  is  an  instance  of  any  of  a number  of 
virtual  "screens"  which  can  reside  in  the 
geography  of  Dataland.  The  type  of  vis- 
ual symbology  or  imagery  surrounding 
any  such  screen  indicates  its  usage:  a 
"drive-in  screen  for  movies."  for  example 
Wnat  we  are  discussing  here  is  a general 
capacity  for  " filmic"  types  of  data,  live 
and  recorded,  which  are  addressable  in 
Dataland  in  user-designated  locales. 
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One  of  the  most  dramatic  visual  experi- 
ences on  our  experimental  Dataland 
surface  is  that  provided  by  LANDSAT 
photographs  of  about  four  thousand 
square  miles  of  the  New  England  coast, 
from  the  Boston  area  up  into  Maine. 

In  the  northwest  corner  of  our  experi- 
mental Dataland,  we  have  an  image  of  the 
northeast  sector  of  the  United  States, 
with  an  inset  area  of  the  New  England 
coast.  Upon  traveling  across  Dataland  to 
this  patch  of  New  England,  it  is  possible 
to  zoom  in  on  any  area. 

The  perusal  of  the  LANDSAT  image  has 
its  own  appropriate  key  map.  It  takes  the 
form  of  a view  on  the  key  map  monitor 
of.  in  this  case,  the  entire  sector  of  New 
England.  The  "key"  consists  of  a high- 
lighted rectangle  moving  around  this 
key  map  image,  tracking  the  course  of 
the  joystick-guided  view  visible  on  the 
large  screen.  Like  the  world  map  view,  it 
offers  direct  teleportation  about  the 
LANDSAT  image  through  touch. 


Zooming  m upon  satellite 
im,un*  ot  Hoston  In  those 
views  appiOiirh  is  from  the 
north  ovei  I'.ipe  Ann  .lint 
M.irhloho.ul  .him  In  first  col 
iimn  top  to  bottom  hlocky 
im.U|e  is  i epl.it  oil  with  finer 
resolution  view  (iimlille) 
wfiile  im.uje  otp.iiuls  .mil 
ti.ivel  proi  eeils  southw.iul 
In  top  .mil  mnlille  panels  of 
tlie  second  column 
.rppio.it  h i ontinues  ovei  the 
penmsul.i  t'f  N.ih.mt  with  v«*t 
.mothei  midije  tepl.n  ement 
1 bottom) 


'M?  MT 


Nearing  Boston,  the  tnangu 
lar  runway  patterns  ot  Logan 
International  Airport  come 
into  sight  at  lower  left  (left 
column)  h mally.  central  Bos 
toil  and  Charles  River  basin 
appear  (right  column,  top 
and  middle) 

To  this  point,  all  images  are 
retrieved  from  magnetic 
media  The  final  image,  a 
map.  replaces  the  satellite 
imagery,  it  comes  off  the 
optical  videodisc 


looming  m upon  Boston  As  tilt'  USl'l  /COINS  111  by  .1  CCrtaill  UlTKHjnt.  Should  WC  .’00111  ill  Oil  tllO  I'llV  Ol  BOStOH. 

the  inctt'asmgly  blocky  material  is  wo  would  find  that  the  I ANI TSAI  imago 

replaced  by  a highet  lesolution  image  in  of  Boston  is  replaced  at  a eeitam  level  01 

perlect  registration  Tins  replacement  altitude  ot  approach  by  a map  ol  Boston 
action  is  performed  a number  of  times  as  in  the  same  si/e  and  registei  as  the 

one  comes  inon  any  portion  ot  the  New  l ANDSA1  image  it  replaces  Then,  as  we 

I ngland  aiea  covered  by  the  l ANDSA1  zoom  in  on  the  Beacon  I till  atea  of  Bos 
image  and  gives  a dramatic  impression  ton,  the  map  image  suddenly  gives  way 
ol  vast  areas  being  contained  within  the  to  a set  ot  slide  views  of  the  central  city 
confines  ol  Dataland  In  the  sequence  ot  area  The  slide  collection  contains  ovei 
pictuies  shown  (coming  ovei  Marble'  200  shift's  ot  Boston  as  seen  fiom  van- 

head  towanf  Boston),  the  final  limit  ot  ios  ous  vantage  points  Tlieie  is  a key  map 

olution  is  tlie  resolution  at  which  the  which  aids  the  usei  in  pci  using  the  col 

original  LANDSAT  matenal  was  photo  lection  of  slides 
giaphed  and  digiti/e’d  I ach  point  ot 
source  data  in  the  stoied  image  repie 
sents  about  an  acre  in  aiea 


Close-up  of  slide  collection 
key  map  used  to  access 
videodisc  still  frames 
Region  along  spine  of 
stacked  slides  ' permits 
sliding  of  cursor  arrow  by 
fingertip  Alternately,  the 
captions  Nent  slide  and 
Previous  slide  can  be 
tapped  tor  oneby-one  viewing, 
or  continuously  touched  lor 
rapid  seguential  showing 


1 ho  interactive  aspect  of  this  key  map  is 
that  the  user  can  slide  a colored-arrow 
' cursor  along  the  top  edge  ot  the  diago 
nal  row  ot  "boxes"  representing  slides 
about  Boston  available  to  be  seen  the 
analogue  tor  more  conventional  media  is 
that  ot  remembering  a slide  tray.  Moving' 
thi'  i uisor  up  and  down  causes  slides  to 
be  (lashed  in  succession  on  the  large 
screen  as  indicated  by  the  finger  touch 
mg  and  moving  the  display  cursor 


t 
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The  active  process,  in  this 
case  a calculator,  is  illustrat- 
ed by  the  photographic 
image  on  the  large  screen 
The  user  can  avail  himselt  ot 
a closer,  touch-sensitive  ver- 
sion. which  can  be  invoked 
as  a virtual  machine.  Other 
examples  might  include,  a 
telephone,  a dictaphone, 
even  a television 


To  recapitulate,  the  first  key  map  was 
the  “table  of  contents"  of  the  "book, 
allowing  the  usei  access  to  selected  set' 
tions  of  the  text  I fit’  second  type  of  key 
map.  exemplified  try  the  time  counter 
dial  associated  with  the  "television  set. 
enables  user  interaction  with  what  aie  in 
essence  processes  over  time  1 lit'  third 
type,  the  LANDSAT  image  key  map. 
functions  with  respect  to  the  LANDSAT 
image  as  the  "woild  view"  auxiliary  mom 
tor  functions  with  respect  to  Dataland  at 
large  a "you  are  here"  marker,  respond 
mg  to  user  joystick  navigation  about  a 
space  The  key  map  associated  with  the 
"slide  collection,"  residing  at  the  map  ot 
Boston  within  the  New  England  LAND- 
SAT image,  exemplifies  the  fourth  type, 
used  to  manage  data  types  which  are 
essentially  characterizablo  fry  "number 
some  ordered  series  of  items,  any  one  ot 
which  the  user  may  dwell  on  foi  some 
unspecified  period  of  time 


We  expect  the  repertoire  ot  key  map 
t\  pes  to  grow  along  with  enlargements  ot 
the  lepertoue  ot  data  t\  ties,  .is  new  data 
types  require  ordering  hv  some  principle 
not  accounted  tor  by  any  previously 
defined  key  map 

t uither  among  data  types  in  Dataland 
that  depart  from  the  usual  are  those  that 
aie  pn'Ct'sst's  That  is,  SDMSs  Dataland 
has  nodes  on  its  surface,  indicated  by 
appropriate  glyphs,  which  make  certain 
functions  available  to  the  usei 

A specious  example  is  the  existence  in 
Dataland  ot  the  image  ot  a hand  held 
calculator  1 he  user  can  zoom  directly  up 
to  it  1 here  then  appears  on  the  small  key 
map  monitor  a copy  of  this  calculator 
image  The  user  can  directly  touch  the 
image  surface  on  the  monitor  to  input 
arithmetic  problems,  just  as  with  an 
actual  calculator  This  particular  example 
of  a process,  perhaps  trivial  or  "cute." 
points  thi'  way  to  what  will  ev entually  be 
an  interesting  and  powertul  class  ot 
data  types 
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When  making  graphical 
annotations  user  enters 
commentary  and  |Ottings 
with  lapboard  and  stylus 


Noted  was  the  idea  that  "perusing" 
implies  the  scrutiny  in  detail  ot  items  Tins 
scrutiny  in  detail  as  an  interactive  proc- 
ess surely  will  trigger  in  the  user  the 
impulse  to  annotate,  to  append  jottings, 
to  comment  "marginally  " The  traditional 
properties  ot  computer  media  typically 
frustrate  this  impulse,  one  so  readily 
indulged  in  on  paper 

As  an  antidote  to  this  shortcoming,  we 
have  introduced  the  capacity  for  making 
written  annotations  in  "transparent  ink." 
of  varying  line  widths  and  colors.  The  ink. 
because  transparent,  does  not  obliterate 
material  overwritten,  and  the  notations 
made  can  be  transient  or  permanent  as 
the  user  wishes 


Notes  and  commentary  can  be  not  only 
in  written  form,  but  auditory  as  well  The 
user  can  employ  the  stylus  and  tablet  to 
indicate  an  "anchorpoint"  on  the  Data- 
land  surface  for  to-be-recorded  voice 
notes  The  notes  are  played  back  subse- 
quently upon  coming  within  some  critical 
range  of  the  site  of  the  recording 

Sound  can  be  itself  a data  type  as  well  as 
a vehicle  for  annotations  or  navigational 
aid.  An  instance  of  sound  as  data  in  our 
initial  SDMS  prototype  was  a digitally 
stored  speech  of  Jerome  Wiesner,  Presi- 
dent ot  MIT  A portion  of  this  speech  was 
activated  and  played  back  upon  approach 
to  an  image  of  Dr  Wiesner.  the  volume 
increasing  as  one  drew  nearer  by  zoom- 
ing in.  or  fading  to  the  left  or  right  as  one 
withdrew  from  face-center  toward  the 
right  or  left,  respectively  The  speech 
sound  was.  in  effect,  synchronized  with 
the  location  of  the  speech 
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An  octaphonic  sound  capability  permits 
ttie  creation  in  the  media  room  ot  a veri- 
table "cocktail  party”  of  sounds  and 
voices,  any  sound  so  re-created  as  to 
give  it  a spatially  definite  apparent 
source  in  ttie  room  Endowing  sound 
sources  with  specific  locales  effectively 
creates  a set  of  "channels."  aiding  sepa- 
ration and  identification  of  a number  of 
simultaneously  heard  voices.  Ttie  possi- 
bility of  "browswig"  through  sets  of 
auditory  messages,  redundantly  cued  in 
visual  graphics,  can  make  sound  as  a 
•lata  type  or  means  of  annotation  a rich 
offering  to  the  user 


‘Ink"  is  transparent,  and 
does  not  obliterate  material 
beneath  Note  ink  color 
options,  and  color-stnp 
highlighting  over  address 
in  letterhead  and  in  saluta- 
tion. Ink  can  be  temporary, 
permanent,  or  placed  in 
logically  separable  layers 


The  ensemble  of  data  types  and  asso- 
ciated interactions  enumerated  here 
form  an  initial  set  and  by  no  means 
exhaust  the  long-term  potentialities  of 
user-system  interaction  Before  we  turn 
to  a consideration  of  what  the  future  may 
hold,  however,  let  us  set  down  some  of 
our  mapr  findings  thus  far  about  using 
space  to  organize  data 


What  We  Learned 


A major  lesson  of  our  experience  in 
attempting  to  use  space  to  articulate 
information  was  the  virtue  of  straightfor- 
ward simplicity  in  the  presentational  as- 
pects of  the  user's  "virtual"  spatial  world 

The  system  as  outlined  above  did  not 
spring  into  existence  all  at  once  but 
resulted  from  many  years  of  Office  of 
Naval  Research  research  in  television 
based  computer  graphics  and  an  initial 
version  of  a spatial  data-management 
system.  SDMS  I.  developed  over  the 
project's  first  year  of  DARPA  support 

SDMS  I was  a "2-1  2-D"  spatial  world  a 
set  of  2-D  data  planes,  so  linked  as  to 
define  a hierarchical,  "laminar"  space. 
The  top  level  of  this  structure  was  Data- 
land  itself:  a large  data  plane  with  "items" 
of  various  sorts  arranged  on  its  surface, 
some  grouped  on  common  background 
colors  As  described  earlier,  the  large 
screen  in  front  of  the  user  functioned  as  a 
"window"  onto  Dataland,  which  appeared 
in  its  entirety  on  the  small  monitor. 


Importantly,  the  scheme  of  the  data 
space  in  SDMS  I included  a system  of 
"ports"  on  any  data  plane  or  layer,  which, 
when  roomed  in  on,  would  give  way  to  a 
next  data  layer,  seemingly  in  back  of  or 
"beneath"  it  In  the  accompanying  illus- 
tration. we  can  see  "ports " leading  from 
the  top  Dataland  layer  to  different, 
second-level  data  planes.  One  of  the 
second-level  ciata  planes  in  the  illustra- 
tion is  a terminal  node,  but  the  other 
second-level  data  planes  themselves 
have  ports,  each  leading  down  to  third- 
level  data  planes  The  information  con- 
tained on  the  lower  planes  was  logically 
subordinate  to  the  topic  at  the  port 
through  which  the  user  dove 

Travel  around  this  system  was  up  and 
down,  chaining  through  the  hierarchical 
structure  of  the  nodes  (ports)  on  the  vari- 
ous data  planes.  One  could  travel  deep 
down  into  a certain  data  structure  and 
then  "chain  back"  through  the  structure 
to  get  back  to  the  point  of  entry. 
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Our  SDMS  I system  was  programmed  to  The  resolution  of  the  dilemma  was  the 

include  up  to  twenty  ports  in  any  plane.  insight  that  both  the  finding  of  items  and 

and  the  "depth  of  the  structure  in  levels  the  navigation  in  data  space  could  best 

was  in  principle  indefinite  As  a practical  be  served  by  a simple,  single-plane 
matter,  in  our  demonstration  version  the  Dataland.  This  sort  of  space  seemed  to 

deepest  depth  in  chaining  down  a hier-  offer  all  the  primary  benefits  of  locatabil- 

archy  was  about  seven  levels.  ity  to  be  derived  from  the  SDMS  principle 

without  having  all  the  disadvantages  of 
1 he  spatial  structure  of  SDMS  I.  however.  complexity  that  a laminar  space  might 
generated  a frustrating  design  paradox  entail 
for  any  comprehensive  data-world  image 

for  display  on  the  "world  view'"  monitor  Thus,  in  our  second  prototype.  SDMS  I!. 

the  nested  laminar  space  of  SDMS  I 
Although  a number  of  candidate  schemes  was  replaced  by  a single,  very  large 
for  mapping  views  were  advanced.  data  plane.  When  approaching  an  item 

including  quite  ingenious  neural  net-like  through  the  joystick  " zooming  " action, 

schemes,  some  serious  drawback  the  user  does  not  “pop  through  a port"' 

seemed  always  to  attend  them  Those  down  to  another  level  of  data  space  to 
that  were  visually  simple  and  easy  to  ODtain  further  information.  Rather,  once 

interpret  were  so  at  the  expense  of  infor-  the  viewer  has  come  up  close  upon  the 

mation  content.  ThosJ  that  were  compre-  item,  the  action  from  nis  or  her  point  of 

hensive  became  "baroque  : the  more  view  is  then  to  "peruse"  the  item  in  place. 

accurately  they  portrayed  the  rather  as  it  were,  with  respect  to  the  greater 

complex  paths  that  some  user  might  take  space  of  Dataland.  That  is.  the  perusal  of 

through  a data  space  consisting  of  any  item  occurs  without  any  sense  of  fur- 

nested  planes,  the  more  the  mapping  ther  "travel"  in  data  space  of  the  sort 
view  violated  the  principle  that  a display  implicit  in  going  down  through  ports  to 
system  be  visually  straightforward  and  deeper  layers  of  data, 
easily  interpreted. 


As  a result,  in  SDMS  II,  locating  an  item,  While  (light  in  simulated  3-D  space  can 
and  perusal  of  that  item  once  located,  be  dramatic  and  compelling,  to  the 
are,  logically,  two  independent  activities.  extent  that  3-D  flight  is  convincingly 
In  SDMS  I,  these  activities  were  con-  simulated,  then  to  that  extent  the  possi- 

tounded  each  with  the  other,  in  that  as  bilities  are  expanded  lor  becoming  con- 
you  looked  at  more  information,  you  also  fused  in  that  space.  Spatiality.  which  in 
popped  through  down  and  deeper  into  the  SDMS  context  is  supposed  to  be  in 
"location  space"  as  well.  the  service  of  managing  data,  can  then 

itself  become  so  rich  in  directionalities 

What  about  an  SDMS  in  3-D,  with  3-D  that  it  generates  sets  of  navigational 
“flight"?  issues  which  in  their  turn  demand 

resolution. 

At  our  protect's  inception,  the  possibility 

of  creating  a spatial  data-management  Beyond  the  issue  of  user  orientation  in 
system  in  interactive  3-D  space  in  the  real-time  3-D  flight  is  the  problem  of  the 

manner  of  a flight  simulator  was  recognition  of  objects  in  a dynamic  3-D 

broached.  The  idea  was  an  intriguing  world.  The  ability  to  approach  any  item 
one:  flying  in  and  around  a space  ol  data  in  a 3-D  space,  from  any  angle,  can 
items  under  the  guidance  of  a variety  of  seriously  compromise  the  ready  mden- 
aviation-inspired  navigational  aids.  For  tification  of  that  item.  Things  simply  do 
practical  reasons,  the  idea  was  not  pur-  not  look  the  same  from  different  angles, 
sued:  acquisition  of  real-time  flight  simu-  The  perception  that  a certain  item  is  the 
lation  hardware,  with  visible  surface  por-  “same"  when  seen  from  varying  aspects 
trayal  of  a 3-D  world  in  color,  would  have  (perceptual  "shape  constancy")  is 
far  exceeded  cost  parameters  for  our  robust,  to  be  sure,  but  it  can  and  will 
project.  But,  apart  from  costs,  there  are  break  down 
factors  intrinsic  to  dynamic  3-D  flight  to 
be  considered  in  relation  to  the  ends  to 
be  achieved  in  an  SDMS. 
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Not  only  does  the  internal  structure  of  a 
figure  affect  its  recognition,  but  the  "top." 
bottom,  and  "sides"  as  determined  by 
the  viewer's  relative  orientation  tend  to 
condition  how  it  is  seen,  and  whether  it  is 
recognized  or  not  For  example,  even  a 
^-D  silhouette  map  of  Africa  turned  90 
degrees  is  much  less  readily  identifiable 
as  when  seen  in  its  usual  north-south  ori- 
entation (Rock.  I . The  Perception  of  Dis- 
oriented Figures."  Scientific  American 
January  1974) 


To  summarize  this  point:  the  special  vir- 
ue  of  a spatial  data  management  system 
ies  not  in  a sense  of  space  and  flight  for 
their  own  sakes  but  in  the  invocation  of  a 
sufficient  sense  of  space  to  activate 
the  native,  highly  evolved  capacity  of 
humans  to  organize  their  perceptual 
experience  spatially.  The  optimal  space 
for  an  SDMS.  then,  is  one  that  prompts 
spatial  "schematizing"  in  the  user,  while 
minimizing  the  potential  for  disorientation 


To  have  all  items  always  "face  front" 
independent  of  the  user's  relative  path  of 
approach  in  flight  is  not  necessarily  a 
cure  for  the  object  recognition  problem 
1 he  appearance  0|  items  taken  individu- 
ally is  only  a part  of  perceiving  where  you 
are  and  what  you  are  looking  at  Inter- 
item relational  patterns,  a significant  con- 
textual basis  both  for  recognizing  things 
and  for  navigating  in  "neighborhoods"  of 
thmgs,  become  effectively  different  pat- 
terns when  a grouping  of  items  is 
approached  from  different  directions 


Experience  indicates  that  the  maiority  of 
guest  users  trying  out  their  hand  at  "fly- 
ing across  Dataland  for  the  first  time 
invariably  depend  on  the  auxiliary  view  to 
navigate,  using  the  large  screen  for 
confirmatory"  glances  that  they  have 
arrived  where  they  set  out  to  go  The 
apparent  aspect  of  Dataland'is  so  large 
Hal  this  auxiliary  view  is  vital  for  success- 
ful navigation,  even  for  the  constant,  well- 
practiced  user  who  may  have,  over  time 
gained  a very  detailed  mental  picture  of 
the  lay  of  the  land." 
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It  turns  out  to  be  the  case  that  navigation 
over  Dataland  witti  tire  joystick  is  so 
eminently  and  inherently  "response  com- 
patible" that  the  novice  user  gains  profi- 
ciency  in  flight  in  well  under  a minute 
One  o I the  prime  attractions  of  such  a 
system  is  the  readiness  with  which  the 
new  user  becomes  a competent  drivei 

Accompanying  the  rapid  acquisition  ot 
"piloting"  proficiency  is  rapid  assimilation 
by  the  novice  of  the  graphic  imagery  of 
the  Dataland  surface  in  a "spatial"  way 
Almost  immediately,  tfie  new  uset  begins 
to  refer  to  items  hv  describing  them  as 
"the  letter  to  the  right  and  north  of  the 
green  square,"  or  "the  yellow  rectangles 
in  the  southeast  corner."  Although,  of 
course.  Hus  way  of  regarding  data  is 
intended  in  the  system,  the  beginning  ot 
discussing  data  in  this  mode  try  novice 
users  is  gratifyingly  spontaneous. 


Beyond  sound  as  a data  type  and  as 
an  annotating  modality,  our  work  with 
sound  as  a "navigational  aid"  has  built 
an  appreciation  ot  its  utility  for  guidance 
and  orientation,  as  well  as  for  its  enrich- 
ing function  as  a redundant  cue  lor  what 
is  going  on  visually.  Those  occasions 
when  our  own  experimental  sound  sys- 
tem has  been  temporarily  down  have 
convinced  us  that  we  would  be  loath  to 
return  to  a totally  silent  data  world.  Con- 
clusion: sound,  well-orchestrated,  is  cm 
cial  A spatial  data-management  system 
must  have  it 

While  we  learned  much  concerning  spe 
cific  design  features  of  our  SDMS  effort, 
realizations  also  emerged  about  further 
potentialities  of  the  spatial  theme.  Some 
ot  these  indications  tor  future  directions 
are  set  forth  in  the  following  section 


Future  Directions 


Progress  in  SDMS  during  our  lirst  two 
years  of  pursuing  spatiality  as  a motit  lias 
encouraged  us  to  chart  a prospectus  ot 
next  steps  that  is  advisedly  extreme  in 
terms  ol  positions  taken. 

Future  directions  elucidated  include  those 
of:  topical  spatiality.  gaze-contingent 
displaying,  eye-gesture  coordination, 
voice-driving  with  eye  contact,  exploit 
mg  the  user's  near-field,  user  position 
sensing  through  a space-sensor  ed  "lab 
lacket " 

The  idea  underlying  "topical  spatiality"  is 
simple:  the  topography  ot  Dataland  is 
organized  on  the  fly.  as  a function  of  usei 
selected  axes.  These  axes  are  chosen 
by  the  user  on  the  basis  of  what  data 
dimensions  are  judged  appropriate  foi 
some  specific  task.  In  other  woids,  unlike 
the  current  arrangement,  pictograms  are 
not  stored  and  left  in  one  place,  but  form 
part  of  a changing  2-D  space 


As  an  example,  consider  data  typical  ol 
a company's  personnel  file  Pictures  of 
employees  would  reside  on  secondary 
storage  media,  and  Dataland  would 
have  an  indicative  but  blank  army  of  /ee 
tangles.  The  user  might  then  speedy  Ins 
horizontal  axis  to  be  a measure  ot  ago 
and  tlic'  vertical  axis  .is  something  like 
years  of  service,  salary  level.  01  manage 
rial  responsibility.  Once  selected,  the  pie 
tint's  would  be  ordeied  on  the  tly  in  tins 
2 D space,  with  overlap  anti  clustering, 
most  probably  1 tit'  data  would  be 
searched  with  the  knowledge'  tlr.it  Mi 
Smith  (tilt'  company's  founding  tallied  is 
somewhere  at  uppei  light  and  the  deliv 
ery  boy  is  in  the  southwest 


We  propose  to  implement  a topical  This  theme  will  include  unclustering  pro- 

SDMS.  within  the  existing  system,  using  cedures  that  provide  the  proper  chang 
building  types  The  selection  ot  data  ' mg.  nonlinear  axes  while  the  user  zooms, 
comes  from  our  54. OOO-ditferent  frame  in  order  to  "spread  out  overlapping 

videodisc  that  was  made  in  collaboration  data.  What  wo  should  loam  are  lessons 
with  the  MIT  architecture  slide  library  concerning  people's  ability  to  create 

and  contains  about  20.000  slides  of  mental  spaces  and  then  to  search  them. 

American  Architecture  At  this  writing, 

those  "frames"  are  additionally  described  A critical  focus  of  further  research 
in  a data  structure  which  includes  state,  involves  a set  of  themes  having  to  do 
county,  city,  architects'  names,  dates  of  with  the  active  engagement  of  the  user  in 
construction,  and  twenty  building  types.  the  environmental  space  ol  the  interface. 
The  data  lend  themselves  to  traditional  expanding  the  spatial  domain  to  encom- 

spatiality  (i  e . they  can  be  viewed  on  a pass  the  real-space  characteristics  ol 

map  of  the  US),  but  they  also  fall  onto  user  actions.  The  overall  intention  is  to 

nonspatial  axes  like  date  of  construction,  render  the  system  cognizant  of  tiow  the 

physical  scale,  degree  of  public  access,  user  is  operating  in  Ins  personal  space, 

and  the  like  In  the  case  ot  date  and  size  as  this  space  intersects  with  the  shared 

(X  and  y.  respectively),  if  we  limit  our  space  of  the  interface:  directionalities  in 

selection  to  post  Civil  War.  we  can  look  looking,  speaking,  touching,  gesturing, 
for  the  World  Trade  Center  in  tire  north-  together  with  a system  awareness  of 

east  and  Grant's  tomb  at  the  lower  left  user  position  and  proximities 

Among  these  themes  is  the  awareness  of 
where  the  user  is  looking  Eye  contact  is 
an  important  adjunct  in  human  interper- 
sonal exchange  and,  in  general,  a prime 


indicator  of  the  focus  of  active  awareness. 


The  opportunity  hero  is  tor,  among  other 
things,  an  eye  controlled  "virtual  cam 
era,"  that  is.  a display  that  points,  pans, 
aooms.  travels,  as  a (unction  ol  where  the 
user  is  looking  In  the  context  ot  the 
imagery  in  SI  TMS  so  tar  discussed,  pos 
sibilities  might  include  even  the  trivial 
example  ot  a spatially  dispersed  set  ot 
items,  perhaps  tacos,  each  ot  which 
'talks"  when  eye  addressed  try  the  user, 
the  dnectod  ga/e  used  by  the  system  as 
an  implicit  "command"  or  act  ot  selee 
tion  Another  example  would  be  an 
image,  say  a map.  which  locally  expands 
and  gams  detail  as  a (unction  ol  the  point 
and  duiation  ot  "looks" 

The  sense  ot  space  in  and  at  the  inter 
face  includes  system  awareness  ot  the 
user  near  field  The  tone  ol  appointments 
in  tlu'  media  room,  especially  the  open 
ness  ol  tht'  chair,  invites  the  user  to  gel 
up  out  ot  tht'  chair  the  more  immediately 
to  engage  himsell  with  the  data  Ges 
taring,  arm  waving,  pointing,  especially 
in  conjunction  with  line  ol  sight  indicn 
lions  captured  through  unobtmsive  eye 
tracking,  would  endow  the  immediate 
space  between  user  anti  system  with 
communicative  meaning 


Space  tanging  techmgues  that  permit 
tlu'  system  to  know  whore  the  user  is  in 
the  space  ot  the  interlace,  plus  the  user  s 
body  attitude,  have  intrinsic  uses  A sys 
tern  til  body-borne  space  sensors,  unob 
tmsively  disposed  about  a soit  ot  "lab 
lacket"  as  buttons,  cutt  links,  01  epaulette 
would  establish  a computet  inter protable 
dot  figure  tlemaikmg  the  momeiitar\ 
position  anti  attitude  ot  the  user  Among 
other  functions,  such  knowledge  cun 
help  to  it'  establish  tracking  t'ontai't  ot  an 
errant  oculomotor,  making  sensing 
where  one  is  looking  a mote  icalistie 
proposition  in  a dynamic  enyiionmont 


With  body  sensing  in  the  user  near-field, 
pushing  items  of  virtual  data  about  in  a 
now-tangible  3-D  space  arises  as  a pos- 
sibility. Or.  traffic-patrol-like  direction  of 
items  streaming  by  in  patterns  for  on-the- 
fly  perusal.  The  dynamics  of  orchestra- 
tion. literally  like  those  of  conducting  a 
symphony,  arise  as  realities,  especially 
pertinent  in  an  environment  of  spatially 
distinct  sounds  and  voices  in  an  octa- 
phomc  surround  Pointing  to  virtual 
talkers  with  indications  to  speak,  to  hurry 
up,  to  pause,  becomes  repertorial  for  the 
user,  now  "fleshed  out"  at  least  minimally 
in  dot  form  in  the  augmented  space  of 
the  interface. 

Voice  direction  with  eye  contact,  a natu- 
ral combination  in  the  extra  system  real 
world,  persist  in  the  realm  of  user-system 
interchange  as  a viable  modality.  The 
ability  to  voice-activate  takes  on  more 
precise  meaning  and  possibilities  in  con- 
junction with  the  ability  to  address  by 
voice  and  eye  simultaneously  in  a coor- 
dinated fashion.  To  be  able  to  point  by 
gesture  as  well  defines  yet  an  additional 
dimension  of  voice-and-eye  dynamics. 


These  themes,  somewhat  synoptically 
expressed,  implicate  actions  in  an 
explicit  space,  coextensive  with  the  real 
room  space  at  the  user-system  interface, 
a shared  space  which  can  be  artfully 
integrated  with  the  virtual  spaces  which 
have  been  the  focus  of  this,  our  work, 
so  far. 

At  stake  here  is  a user-system  interface, 
a data  space,  so  enriched  and  quick- 
ened that  the  human  becomes  more 
than  a dimensionless  point,  the  space  at 
the  interface  something  other  than  empty 
volumes.  The  long-term  goal  is  the 
utmost  in  usability,  where  the  user's  nat- 
ural modalities  and  capacities  for  commu- 
nicative expression  meet  with  perceptive, 
complementary  system  response. 
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